Controlled perspective guidance method

ABSTRACT

Navigation through a network of body passages, such as through the airways, is improved by a method of displaying information which provides a fly-through visualization of the passageway. As landmarks or waypoints are passed, the view changes to a next segment in a planned pathway to a target. Such a visualization technique avoids tunnel vision, such as that encountered while using real-time endoscopes.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.12/369,466, filed Feb. 11, 2009, which claims priority to U.S.Provisional Application Ser. No. 61/028,098 filed Feb. 12, 2008 entitledControlled Perspective Guidance Method, which is hereby incorporatedherein by reference.

BACKGROUND OF THE INVENTION

Identifying and treating lung tissue abnormalities presents challengesthat are somewhat unique to the lungs. If a tissue lesion or tumor is tobe identified and excised surgically, the chest wall must be opened toprovide access to the lungs. Opening the chest wall is a commonprocedure but one that presents risks of infection and lengthy recoverytime, nonetheless.

A desirable alternative to surgery, in terms of reducing patient trauma,is to identify and excise the tumor endoscopically. Endoscopic surgeryin the lungs, however, means that the complicated bronchial maze must benavigated. Endoscopes have cameras at their distal tips that provide aphysician a real-time image through the end of the endoscope while theendoscope is being advanced through the bronchial system. However,typical endoscopes are too wide to be advanced deep into the lungs asthe diameters of the airways decrease toward the alveoli.

In order to assist in navigating deep in the lungs, systems have beendeveloped that include a sensor at the distal tip of a probe. The sensorinteracts with a field generator to provide a real-time indication ofits location in a three-dimensional space encompassing the lungs. Thereal-time indication is superimposed on a computer image of thebronchial tree (known as a “virtual bronchial tree”) so that thelocation indication generated by the sensor is useful to the physician.

Heretofore, the location data has been presented to the physician asthough the sensor is actually an endoscope. Hence, the virtual bronchialtree image is presented from the perspective of the tip of the sensorand as the sensor is advanced, the image of the virtual bronchial treemoves past the sides of the sensor and disappears.

The difficulty when viewing navigation progress in this manner, as istrue with an endoscope as well, is that the physician has tunnel vision.No peripheral vision is available. Thus, if the end of the sensor isadjacent a branch, the branch may not appear on the screen. Thephysician wanting to see to the sides of the endoscope or, in thevirtual sense, the sensor, must either retract the probe or turn it inthe direction of the branch.

This visualization problem becomes even more confusing when consideringfirst that the sensor is moving with the cardiac rhythm and breathingcycle of the patient and second that the sensor often twists when thesteering mechanism is activated. The cardiac rhythm and breathing cyclecause jittering of the virtual image, which can be very disorienting.With regards to the use of the steering mechanism, most steerable probeshave a tip that can be turned in one to four directions. This means thatin order to turn the tip in a desired direction, it may be necessary torotate the tip around its longitudinal axis up to 180 degrees prior todeflection. This will cause the image being viewed to flip and thenturn. The physician can easily become disoriented after a few turnswhile trying to navigate to a lesion. There is a need to provide moreuseable data display for use in navigating a probe through a virtualbronchial tree.

SUMMARY OF THE INVENTION

The system and method of the present invention provides more useablevisual presentation to a physician using an intra-body navigationsystem. Generally, the system and method of the present invention uses asystem of planned waypoints along a planned route through the bronchialtree in order to incrementally advance the image presented to thephysician.

Thus, a physician using the system and method of the present inventionwill see an image of the bronchial tree and an image of the sensor beingadvanced through the airways to a subsequent waypoint. Once the sensorhas reached the next waypoint and has been turned down an appropriatebranch, the image perspective will be advanced to that next waypoint.Rather than the images being presented as though there were a cameramounted on the sensor, which may be moving in a chaotic, erraticfashion, the images of the virtual bronchial tree are presented in acontrolled manner, for example as though cameras are mounted on each ofthe waypoints.

In one embodiment, rhythmic movement (movement caused by the breathingcycle, cardiac cycle, or patient movement) is visualized by tying theperspective of the video image to the visible portion of the bronchialtree such that there is no viewable relative movement between the“camera” and the bronchial tree. Rather, all rhythmic movement isvisualized by showing movement of the probe. Hence, if the bronchialtree moves in such a manner that it affects the relative position of theprobe within the airways, the probe is seen as moving. This provides thephysician with a more accurate perception of the location of the probe.

In another embodiment, calculations and visualization are simplified byincorporating the reality that while the probe is being advanced througha section of the airways that contains no branches, the probe will beautomatically guided by the sidewalls of the airway. Hence, noparticularly useful information is gleaned from an image that is showingrhythmic movement. In this embodiment, a simplified view of the sensorbeing advanced through the center of the airway is provided until thesensor reaches a fork. Once the fork is reached, the probe will be shownas veering down one of the available branches. In this way, thephysician knows which branch the probe has entered. A preferred aspectof this embodiment provides a highlighted representation of the targetbranch, as determined in a pre-operative planning stage.

In another embodiment, the airway is represented by a simple tunnel withthe next waypoint or branch highlighted somehow. As the probe isadvanced, the highlighted waypoint increases in size to simulate gettingcloser. As the probe appears to pass the highlighted waypoint, the viewswitches, either manually or automatically, to a new airway with thenext waypoint highlighted in the distance.

In yet another aspect of the present invention, an overlay or secondwindow is displayed giving a schematic diagram of the bronchial tree andthe entire path to the target shown. Preferably, an indication of thepresent sensor location is shown, thereby providing the physician withan indication of the probe's progress towards the target.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an example of a location system that may be usedwith the present invention; and

FIGS. 2-22 are examples of displayed images of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Referring now to the figures and first to FIG. 1, there is shown alocation system 10. Though the navigation display system and method ofthe present invention may be used with any location system havingvirtual capabilities, a description of a location system 10 is providedby way of example.

The location system 10 generally includes a locatable guide 20, alocation board 40, and a control system 80. The locatable guide 20 is aprobe having a receiver that generally includes a plurality of(preferably three) field component sensors 22, 24 and 26. Each of thefield sensor components is arranged for sensing a different component ofan electromagnetic field generated by the location board 40.

The location system 10 also includes the location board 40. The locationboard 40 is a transmitter of electromagnetic radiation. The locationboard 40 includes a stack of three substantially planar rectangular loopantennas 42, 44 and 46 connected to drive circuitry 48. Drive circuitry48 includes appropriate signal generators and amplifiers for drivingeach of the loop antennas 42, 44 and 46 at their correspondingfrequencies. The electromagnetic waves generated by the location board40 are received by the locatable guide 20 and converted into electricalsignals that are then sent to the control system 80.

The control system 80 generally includes reception circuitry 82 and adisplay 86. The reception circuitry has appropriate amplifiers and A/Dconverters. The reception circuitry 82 and the driving circuitry 48,which may be considered part of the control system 80, are controlled bya controller/processor 84 that typically is an appropriately programmedcomputer. The controller/processor 84 directs the generation oftransmitted signals by driving circuitry 48. The controller/processor 84also generates video signals, which are sent to the display 86.

Having introduced the basic system 10 on which the present invention ispracticed, attention can now be focused on the manner in which thecontroller/processor 84 presents information on the display 86.

Preferably, all of the embodiments of the present invention have theability to display an overview of the bronchial tree. This overview maybe obtained from subjective sources such as CT images or objectivesources such as medical atlases, etc. An example of such an overview 100is provided in FIG. 2. The overview 100 may be displayed on a splitscreen or on a separate monitor. The overview may also be presented in aseparate window on the display 86 but it is likely that a physicianwould benefit from being able to constantly monitor the overview 100.The overview 100 includes an indication of a targeted lesion 102 and apath 104 to the lesion 102. The path 104 is determined in apre-operative planning stage, either manually or automatically, or acombination thereof (semi-automatically, for example).

The path 104 includes a plurality of waypoints or turning points 106 a-denroute to the lesion 102. Navigation to these waypoints 106 a-d issupplemented by identifiable anatomical landmarks 107 a-e, typicallybifurcations, trifurcations, or forks.

During navigation to the target lesion 102, an virtual interior view ofthe airway will be displayed along with the overview 100. This interiorview may be realistic, constructed from CT scans, for example, or may bevery basic (schematic). Additionally, this may be presented as aperspective or parallel geometric view (i.e. with or without a vanishingpoint). Hence, in a preferred embodiment, FIGS. 3, 5, 7, 9, 11, 13, 15,and 17-22 show an interior display which will be presented with theoverview 100. Various embodiments of the present invention will bedemonstrated using the same path 104 and target lesion 102 from FIG. 2.

A first embodiment is shown in FIGS. 3-16. In this embodiment, thedisplay is shown from a viewpoint just over or beyond the distal tip ofthe locatable guide. The display remains centered in the airway and doesnot twist or otherwise move with the movement of the locatable guideexcept in an axial direction. FIG. 3 shows the first landmark 107 a andthe first waypoint 106 a. Because the waypoints are turning points andnot (necessarily) physical landmarks, the waypoints may be displayed assuspended points that indicate to the physician that upon reaching thewaypoint, the physician needs to steer the probe in the next direction.Alternatively, the waypoints need not be displayed. In order to easenavigation, the correct airway will be highlighted somehow, as shown.Optionally, the highlighted airway could change color or blink when theprobe reaches the waypoint to indicate a direction change is needed. Tofurther assist with navigation, the position of the locatable guide isshown as an arrowhead 108, or some other indication, on the overview 100as shown in FIG. 4.

FIG. 5 shows that as the locatable guide is advanced the features of thebifurcation marked as landmark 107 a grow larger, indicating that thelocatable guide is nearing the waypoint 106 a. The airway along the pathremains highlighted to assist the physician in navigating to the lesion102. Also, as seen in FIG. 6, the arrowhead 108 continues to advancealong the path. It is envisioned that the arrowhead 108 may always bedepicted along the desired path, thereby providing only an indication ofhow far along the path the locatable guide has been advance, or morepreferably, the arrowhead may float independent of the path, therebyproviding indication of the location of the locatable guide in the eventthat the physician has advanced the tip of the probe down an incorrectairway.

After the probe has passed through the opening of the correct airway,the view changes to that shown in FIG. 7, with the next waypoint 106 bhighlighted and an adjacent landmark 107 b in the distant field view.Optionally, a more distant bifurcation 110 is depicted, giving depth tothe perspective view of the airway. This bifurcation labeled forreference purposes in FIG. 8, which corresponds to the view in FIG. 7.The arrowhead 108 is still visible in FIG. 8 but has not advancedconsiderably past its position in FIG. 6, because crossing the thresholdof a waypoint to a different view may be accomplished with a minuteactual advancement of the locatable guide.

FIG. 9 shows the steady growth of the features 110, 107 b and 106 b asthe locatable guide is advanced. FIG. 10 shows that the arrowhead 108 isclose to approaching the waypoint 106 b.

Upon passing waypoint 106 b, the view changes to that shown in FIG. 11.Landmark 107 c is visible with the correct airway containing waypoint106 c highlighted. FIG. 12 is the corresponding overview 100 witharrowhead 108 located just past waypoint 106 b and pointing in thedirection of the next waypoint 106 c.

Upon passing landmark 107 c, the view changes to that shown in FIG. 13.Landmark 107 d is visible with the correct airway highlighted. Notably,waypoint 106 c has not yet been reached and still appears in thedistance. This illustrates that landmarks may be used as trigger pointsto update the views. This is useful for long straight passages thatresult in passing several anatomical features. Alternatively, onlywaypoints may be used as trigger points. FIG. 14 shows that thearrowhead has advanced past waypoint 106 c.

FIG. 15 provides the view after passing landmark 107 d and correspondswith the overview 100 shown in FIG. 16. Optionally, the target 102,which is now visible, is highlighted in a different color or patternthan the various waypoints. It is also preferable to provide anindication when the tip of the probe is within an operational proximityto the target 102, such that the physician knows that the appropriatetask (biopsy, ablation, excision, etc.) may be performed without furtheradvancing or orienting the probe.

Another presentation embodiment is shown in FIGS. 17-22. In thisembodiment, rather than indicating the advancement of the probe byproducing a growing effect for the various features, a virtual image ofthe probe itself is provided and the viewpoint remains fixed just beyondeach waypoint. Optionally, the physician may be given the option toadvance the waypoint manually, in the event that two waypoints arespaced apart such that the image of the next waypoint is difficult tosee.

FIG. 17 shows the first landmark 107 a and the tip of the locatableguide or probe 112. In order to ease navigation, the correct airway willbe highlighted somehow, as shown. To further assist with navigation, theposition of the locatable guide is shown as an arrowhead 108, or someother indication, on the overview 100 as shown in FIG. 4.

FIG. 18 shows that as the locatable guide is advanced the features ofthe bifurcation marked as landmark 107 a remain the same size, but moreof the probe 112 becomes visible, as though the probe 112 is advancingpast and away from the viewpoint. The probe 112 is approaching thewaypoint 106 a and the physician can see that the probe 112 is turnedtoward the correct airway. Also, as seen in FIG. 6, the arrowhead 108continues to advance along the path. It is envisioned that the arrowhead108 may always be depicted along the desired path, thereby providingonly an indication of how far along the path the locatable guide hasbeen advance, or more preferably, the arrowhead may float independent ofthe path, thereby providing indication of the location of the locatableguide in the event that the physician has advanced the tip of the probedown an incorrect airway. Preferably, as shown in FIG. 19, in the eventthat the probe 112 is advanced down an incorrect airway, the viewpointwill not change to the next viewpoint. Rather, the probe 112 remainsvisible and is shown as advancing down the incorrect airway. Thisprovides the physician with an easily understood indication that theprobe should be retracted and advanced down the illuminated orhighlighted airway. Preferably, the virtual probe 112 is constantlyrepresentative of the position and orientation of the actual probe.

After the probe has passed through the opening of the correct airway,the view changes to that shown in FIG. 20, with the next waypoint 106 bhighlighted in the distant field view. Optionally, a more distantbifurcation 110 is depicted, giving depth to the perspective view of theairway. This bifurcation labeled for reference purposes in FIG. 8, whichcorresponds to the view in FIG. 20. The arrowhead 108 is still visiblein FIG. 8 but has not advanced considerably past its position in FIG.19, because crossing the threshold of a waypoint to a different view maybe accomplished with a minute actual advancement of the locatable guide.

FIG. 21 shows the advancement of the probe 112 to a point where theprobe may be steered toward the highlighted airway. Preferably, as theprobe is steered, the virtual probe 112 is depicted as deflecting in thedirection that the actual probe is deflected. FIG. 10 shows that thearrowhead 108 is close to approaching the waypoint 106 b.

FIG. 22 provides the view after passing waypoint 106 c and correspondswith the overview 100 shown in FIG. 16. Optionally, the target 102,which is now visible, is highlighted in a different color or patternthan the various waypoints. It is also preferable to provide anindication when the tip of the probe is within an operational proximityto the target 102, such that the physician knows that the appropriatetask (biopsy, ablation, excision, etc.) may be performed without furtheradvancing or orienting the probe. Alternatively, as shown in FIG. 22,the probe 112 itself is highlighted to show that the probe is inoperational proximity to the target lesion 102.

Although the invention has been described in terms of particularembodiments and applications, one of ordinary skill in the art, in lightof this teaching, can generate additional embodiments and modificationswithout departing from the spirit of or exceeding the scope of theclaimed invention. For example, it is envisioned that the controlledperspective imaging method of the present invention could be used with asystem that uses a navigation device other than a locatable guide, suchas an optical navigation system. Using inputs such as identifiablelandmarks and bronchoscope depth, through image processing the displaymethod of the present invention could be utilized to provide a moreuser-friendly navigational interface. Accordingly, it is to beunderstood that the drawings and descriptions herein are proffered byway of example to facilitate comprehension of the invention and shouldnot be construed to limit the scope thereof.

1. (canceled)
 2. A method of assisting navigation of a probe through abranched network of body lumens, comprising: monitoring movement of aprobe using a three-dimensional tracking system; and displaying a userinterface on a monitor, the user interface including: acomputer-generated image of a distal portion of the probe and a virtualimage of an interior of a lumen as seen from a viewpoint within thelumen; wherein: as the probe moves from a first location within thelumen to a second location within the lumen, the virtual image of theinterior of the lumen changes and the computer-generated image of thedistal portion of the probe remains fixed; as the probe moves from thesecond location within the lumen to an anatomical landmark, the virtualimage of the interior of the lumen remains fixed and thecomputer-generated image of the distal portion of the probe changes; andwhen the probe arrives at the anatomical landmark within the lumen, thevirtual image of the interior of the lumen changes to a differentvirtual image associated with the anatomical landmark, the differentvirtual image remaining fixed as the probe moves beyond the anatomicallandmark.
 3. The method of claim 2, wherein displaying a user interfaceon a monitor further includes illuminating a next lumen along a pathwayto a target.
 4. The method of claim 2, wherein displaying a userinterface on a monitor further includes illuminating a target.
 5. Themethod of claim 2, wherein displaying a user interface on a monitorfurther includes illuminating a waypoint.
 6. The method of claim 2,wherein displaying a user interface on a monitor further includesdisplaying a schematic image of an interior of a lumen.
 7. The method ofclaim 2, wherein displaying a user interface on a monitor furtherincludes displaying a perspective image of an interior of a lumen. 8.The method of claim 2, wherein displaying a user interface on a monitorfurther includes displaying a parallel geometric image of an interior ofa lumen.
 9. The method of claim 2, further comprising displaying asecond user interface on the monitor, the second user interfaceincluding an overview of the branched network of body lumens with anindication of a location of the probe thereon.
 10. The method of claim2, wherein monitoring movement of a probe using a three-dimensionaltracking system includes monitoring movement of the probe using anelectromagnetic tracking system.
 11. The method of claim 2, furthercomprising: defining a pathway to a target and a plurality of waypointsalong the pathway; and updating the viewpoint whenever the probe movespast one of the waypoints.
 12. A system for navigating through abranched network of body lumens, comprising: a control system configuredto determine a position of a probe within the branched network; and amonitor configured to display a user interface, the user interfaceincluding: a computer-generated image of a distal portion of the probeand a virtual image of an interior of a lumen as seen from a viewpointwithin the lumen; wherein: as the probe moves from a first locationwithin the lumen to a second location within the lumen, the virtualimage of the interior of the lumen changes and the computer-generatedimage of the distal portion of the probe remains fixed; as the probemoves from the second location within the lumen to a an anatomicallandmark, the virtual image of the interior of the lumen remains fixedand the computer-generated image of the distal portion of the probechanges; and when the probe arrives at the anatomical landmark, thevirtual image of the interior of the lumen changes to a differentvirtual image associated with the anatomical landmark, the differentvirtual image remaining fixed as the probe moves beyond the anatomicallandmark.
 13. The system of claim 12, wherein the user interface furtherincludes an illuminated indication of a next lumen along a pathway to atarget.
 14. The system of claim 12, wherein the user interface furtherincludes an illuminated target.
 15. The system of claim 12, wherein theuser interface further includes an illuminated waypoint.
 16. The systemof claim 12, wherein the user interface further includes a perspectiveview of the lumen.
 17. The system of claim 12, wherein the userinterface further includes a geometric parallel view of the lumen. 18.The system of claim 12, wherein the user interface further includes acomputer-generated image of an overview of the branched networkincluding an indication of a location of a distal tip of the probe. 19.A user interface for navigating through a branched network of lumens,the user interface comprising: a computer-generated image of a distalportion of a probe and a virtual image of an interior of a lumen as seenfrom a viewpoint within the lumen; wherein: as the probe moves from afirst location within the lumen to a second location within the lumen,the virtual image of the interior of the lumen changes and thecomputer-generated image of the distal portion of the probe remainsfixed; as the probe moves from the second location within the lumen toan anatomical landmark, the virtual image of the interior of the lumenremains fixed and the computer-generated image of the distal portion ofthe probe changes; and when the probe arrives at the anatomicallandmark, the virtual image of the interior of the lumen changes to adifferent virtual image associated with the anatomical landmark, thedifferent virtual image remaining fixed as the probe moves beyond theanatomical landmark.
 20. The user interface of claim 19, furthercomprising an illuminated target.
 21. The user interface of claim 19,further comprising an illuminated waypoint.